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THE GENERATE VCKJ) JOHWSOtMJEYHAH PROOKOUFES: 
AH APPROACH TO COVAhlATK ADJUSTMENT 
AND INTERACTION ANALYSIS 



The purpose of this paper is to describe statistical 
methods for diagnosing and treating three important problems 
in covarlate tests of sign! ficonce--curvi linearity, covariable 
effectiveness and treatment-covariable interaction^ , 

A s sunpt i on s S upport in g Cova ria to A na 1 ysos 
Some recent articles have described the fundamental 

p 

assumptions of most covariate model s' but they neglected to 
explore the relative importance (or unimportance) of each 
assumption especially when applied to quasi-experimental designs^, 
Exploring the relative importance of the assumptions is crucial 
because the qunsi-experiment.nl design places added demands and 
stresses upon the analysis, While the reason for employing a 
covariate tost in experimental designs j s to improve statistical 
precision {rarely of critical concern), the reason for employ- 
ing a covariate tost in quasi -experimental designs is to adjust 
for unknown group biases duo to non-random assignment (alvrsys 
of critical concern). 
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In brief, the major assumptions behind a coveriate 
procedure are the following: 

1. Normality~-criterion and covariables are assumed 

to be normally distributed, 

2. Homogeneity of variances-~the variances of 

criterion and covari ablos are assumed not to 
differ among groups. 

3. Covariable-group i ndonendonco-- the groups are 

assumed to be drawn from a single underlying 
population, and each group reflects the popu- 
lati on covariablc-depondent variable rolati on- 
ship . 

1| . Reliabili ty-**tho covariables are assumed to be 
free of measurement error* 

5>, Linearity- -the covari ablcs are assumed to be 
linearly related to the cri tori on. 

6. Homogeneity of rcr,: , osoion--thc group regression 
equations are assumed to bo independent of 
treatments^ • 

Each of these assumptions will now be discussed in detail. 



Normality and Homogeneity of Variance 



Happily it has been shown that Analysis of Covariance 

(hereafter colled ANCO'M) is robust to violations of normality 

and homogeneity of variance for experiments and cuasi~cxperi - 

ments \tuloss the deviations among groups are bizarre, and 

therefore neither of those assumptions need be of critical 

^ 6 

concern ’ . 
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Co variable -Group Independence 
Covariable-group independence is a sine qua non for 

7 

quasi -experimental covariate analyses « A situation v/bich 
may seem to refute this assumption is the assignment of 

o 

subjects to groups entirely on the basis of covariable scores . 



FIGURE 1 

THE ARBITRARILY PARTITIONED 
SINGLE POPULATION 




As shown in Figure 1, this design is most appropriate when 
a scarce commodity (like academic scholarships) is dispensed 
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on the basis of previous Achievement. The differential 
performance represented by £ is a measure of the extra effect 
of the scholarship beyond the impact of the oovariable. The 
power of this analysis depends on the amount of data near the 
cut points and, consequently, on the continuity between the 
group regression lines. For that reason the example of 
Figure 1 is an ideal situation since t-ho data is most plentiful 
pet the cut point. The success of this analysis depends on the 
fact that the groups represent distinct segments of a s 3 ngl o 
nor mally dis tr ibut ed po pulatio n so that the assumption is not 
violated (only stretched a little). 

Figure 2 illustrates a hypothetical example of Lord's 



FIGURE 2 

LORD'S ANCOVA PARADOX 
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about the problems which can result when this assumption is 
9 

violated , A researcher attempting to contrast the effect 
of two diets makes his initial weight measurements and then 
assigns one diet to a boys dorm and the other to a girls dorm. 
Although the initial and final weight distributions of both 
groups arc identical, ANCOVA would lead to the conclusion that 
boys gained more than girls. Skipping the statistical artifacts, 
it is clear that criterion and covariable have been hopelessly 
confounded with group membership (boys as a groxip outweigh 
girls) o The point of this discussion is that in a quasi- 
experiment it must make sense to equate the groups, i , c , the 
groups must represent the same basic population. 

Reliability 

Although errors in measurement are important, they are 
\isunlly beyond the control of the researcher. It has been 
shown that covariablo reliability levels abovo ,7£ are 
sxifficient. for most situations although a reliability estimate 
can always be used to improve the precision of the analysis^. 

As with all the assumptions that follow, reliability is far 
more critical an issue for quasi-experimontal designs th^n 
for true experiments. 
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Curvilinerrity 



In principle , curvi linearity should not bo a problem 
since all covariate models can bo easily extended to include 
nonlinear terms. In practice, the problem of detecting 
curvilinear! ty and then systematically testing alternative 
regression models requires a good deal of effort. The much 
cited practice of " eyoballing" sc.attcrplots though intuitively 
appealing is just not reliable enough for most analytic 
purposes. A more effective detection method utilizes tests 
for fit and departure from fit 11 . If these tests oro 



incorporated into a stopwiso model (say an 
polynomial ) then terms can be added until 
significant and the departure from fit is 



increasing 
the fit is most 
not significant. 



Figure 3 illustrates this stepwise analysis applied to 
two variables of teacher performance where the best fit has 
been identified as a cubic polynomial . After the best 



FIGURE 3 



A STEPWISE TEST FOR CURVI LI N BARIT Y 



Stop 1: y = 

Source 


b + 
o 

df 


b 1 x; r - -.7?C>1 
ANOVA Tablo 

Sum of So?, Moon Sn. 
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Re "re? r.i on 
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s'p.Mib' 




2.713 
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Pspnrturc 
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ljli^.079 
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1.919 


.3 2? 


Krrors 




E70.S83 
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Total 
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FIGURE 3-- Continuod 

Step 2: y - b Q + b^x + b,->x^ ; Eta = .339 

All OVA Table 



Source 


df 


Sum of Sas 


. Mean So, 


F . 


V. 


Regression 


2 


88.177 


84.039 


9.981 


.138 


Departure 


11 


409.347 


37. .913 


1.999 


.124 


Error? 


14 


270.903 


19.397 






Total 


27 


768,107 








Step 3: y = 


b o + 


P 

b^x hpX + 


b^x ; Sta = 


.479 








ANOVA 


Tab] c 






Source 


df 


Sum of Sqs 


. Mann Sq. 


p 


P 


Regression 




176 . 03 ? 


98. 6?8 


3.036 


. 06 I 1 


Departure 


10 


321.489 


39.149 


1.663 


.186 


Errors 


14 


270,983 


19.327 






To tal 


27 


768.107 









statistics.! node! has been identified in this way, it is 
important to 11 think through 11 tbo relationship and square it 
with the theoretical framework of the study, Tho justification 
of a model chosen by an arbitrary procedure like this lies in 
its shape, not its order* For example, although the fit between 
age and income in Figure k is clearly curvilinear, the 

quadratic equation should be viewed as only a good first 

i P 

approximation to the true relationship*' " , 




<S 



0 



ric;uRE 4 

EXAMPLE 01*’ A CURVILINEAR FIT 
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Sufficiency an-i Efficiency 
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Although sufficiency and efficiency are not assumptions 
made by covariato analysis they pose special problems for 
qua si -experimental designs . Suffi cioncy essentially depends 
on tho ability of the researcher to identify covariablcs which 
account for e very major bins which o?'sts because of nonrandom 
assignment to groups. Efficiency, on the other hand, is 
important bocauso covariate methods arc unusually susceptible 
to inaccuracies due to redundancy among the covari^Mcs, In 
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fact, if two perfectly correlated covariables are used tbo 

1 3 

calculations will completely break down « Once a sufficient 
sot of co variables is available tv/o procedures can bo used 
to locate and remove redundancy* First, all oovarirble pairs 
can be prescreened to identify pairs where the gain from the 
second covariable is less then a specified amount (say 10 °/o } 
of its zero order contribution, When a problem pair is 
identified, the least meaningful covariable can be deleted or 
a new variable can be created combining both variables^. 
Second, a stepwise regression can be performed groupwise co 
identify the efficient set for each group* To be consistent 
in this analysis all terms of a curvilinear covariable should 
be added in a single step* Finally, tbo union of the efficient 
covariable sets for each group is taken (us the covariable set 
for the analysis. As with curvilinear?. fcy, the final sot of 
covariables must be related to the theoretical model to insure 
that results will be j nterpretcblo and meaningful. 

Homogeneity of Regression 

V/h en the criteria described above have been satisfied, 
the threat of nonhomogonoi ty of regression still remains* 
Although ignored by many critics of ANCOVA, a powerful series 
of t o 3 1 s was developed by Palmer 0, Johnson and Jersey Neymen 
to detect treatment-regression interaction? and perform the 
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required tests of significance for two groups and two 
1 £ 

covariables . Vlnet follows is the roathomfci c&l derivation 
of the author’s generalization of these tests to cover any 
number of covariables and groups * 

Assume k groups, n efficient covaria.b? es (>^ , x ) 



and a group regression vector Bj " (b . > K.,, • »., b j )*^* A 

sot of four useful hypotheses can be established as follows: 

H : B-, B 0 */- 4 B } ; each group has a unique 

regression vector. 



H, : B. a B 

1 • X *— i 



, , , ~ P v ; all groups have a common 
regression vector, 

V Ai “ Lo = ... = A k , bj - b-> / ... / b k ; ell 

groups have a cojmmon within group regression 
and different group moans. 



H, : Bj m.. f ^ X ^ = X “ ... = X B ]r ; 

oach group has a uni quo regression vector and 
the groups do not dTffer at the point X, 



Then tho following sot of powerful tests can be employed: 



Test 1: Are there anjv significant differences among 

grouos (JC vs H )? 
x 1 n 



Calculations: Proceeding from the general linear 

model Y - X + e whore X - (1 , , 0 , , , 

and B - (b . . b. . , b .) for each of the 

— oi n m 

k groups, then under the mriypm likelihood 

estimate B. - I?.T. ^ whore R. - ,X. , j and 

the sum of souaros deviation from H is 

■*■ n 



«, »< - 

. V ?/ y £ 

i-1 ih Oj 



iixa.1. 
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where S 
Under II 
% - R T 



p p 

i s distributed 
a 



1 

-1 



f=i N 

the maximum likelihood 



vrhere R = X R. and 
i-1 - 1 



^ - k ( n + 1 ) | . 
estirus te 

k 

T " X T. and 



the sum of squares deviation from is 






k 



« K i ? 

Y ij 



/isj is distributed 



R B. ASj - Sj - Sp end 
S3 X ? [ (k .. I)(n + 1)3. 



AS 



? 

1. 



represents the 3 norease^ in Jbhe sum o f sq uares 

deviations duo to H-. , H, vs H can be tested 

1 1 a 



by forming 



^ s i /df i 

si/ At 

a a 



k 

where df. = (k - l)(n + 1) and df = X N, - k(n + 1 
1 0 i-1 3 



If F^ is statistically significant, then 
it will be fruitful to proceed with the analysis. 
If is not statistically significant the 
analysis can bo terminated since there oro tio 
s i gnlg ice.nt diff e rences in gro u p nog n or 
rogressi ons . 



Test 2: Assuming differences among greu^ are there 

any si gnTlTc^n t7 dilrTer once s among tT) c covnrl able 
coefficients among grouos (H,. vs II )? This test 

is commonly referred to as the test f o r 
homogeneity of regression. 



Calculations: H.. requires a partitioning of the group 

d 
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B, vectors into the group ween b Q . and a cordon 



vn. 



1 thin group regression jA, i.o. = (b^. , JA ) . 
Under Up tho maximum likelihood estimate of 

5}j = ^ 0 i J b ok’ is -H = -H ~H whGrc 



«H = 



( X , Xp , • • « i 


X ) 
n 


f Hi 


•-k 


{? Y . . j .. 


. 2 


\J=1 1 -1 •• 


• 3= 



k Ni 



T 

~H = 



— 








— 


N, 0 .. 


*0 




> 


2ij 


1 






j=i 


0 N 


.0 






• % 

• * 






• 




6 o . . 


• N k 




% 

y 


■7 








5f«i 


/? • • 






ilk . 


V 


Mi , 


2 Z* 

j=l “ x 3 


1 1 » 


X ?/ . 


2 

i=l 


>; , z' . 

j=i ~i j -u 



Likewise the sum of squares deviation due to Up is 



P k N 

s 2 = f Bl 5=1 *ij 

4 



1 Y? , - R„ 



H 2 h* ~ S 2 " S a vrhere 



L ^Sp is distributed a [n(k - 3 .)], and AS^ 



represents the gain in sum of squares deviations 
due to Hp* H p vs 11 ^ con bo tented by 



S^/dfp 

^ S^/df~ 

a a 



where dfb, = n(k - 1 ). 
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If Fp is not eta ti sti cally significant 
then Fischer' o AFC OVA may be employed. If F^ 
is statistically significant then on alter na tive 
to ANCOVA must be empl oy od * 



Test 2k: Given that the covariable coefficients are the 

seme for all groups, do the group means 
differ (H^ vs H 0 )? This is Fischer's Analysis 

of Covariance, It should be used only when 
F^ i s not statistica ll y significant . 



2 2 2 2 

Calculations: AS = S£ - S p where AS is distributed 

p p 

*X‘(k-l) and AS represents the gain in sum of 

squares deviations clue to H* over JU. IL vs 

x c. x d 

can be tested by 



F 



2 a 



&r/(k - i) 



s 2 /dr 2 . 



where df ^ 



y. H, - (k + n). 
i=l i 



If Pp was significant then this test 

should be ignored. It has been shown that 
applying F 2n when Fp is statistically signif- 
icant consistently errodes the power of ANCOVA. 
and generally leads to a finding of 11 no 
significant differences 11 ^-7, if F,, is not 

signi f leant then one of the two follovjing 
alternatives applies: 

1. If Fp p is statistically significant, 

the groups differ according to the 
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wept* s or arnup 
in tore slo, * ~ 

? . If P p D T n not statistically signif« 

icsnt, then the adjusted group means 
do not differ significantly. 

Figure 5 illustrates the importance of 
using the adjusted group me nan in finally 
deciding how the groups differ. .Although the 



FIGURE 5 

ADJUSTED KEANS VS ACTUAL MEANS 



Cri terion 





mean of Group ?. on the criterion is higher 
than that of Group I (by u units), it is clear 
that, if they nro equated on the c-ovariahlo 
then Group 1 would significantly outperform 
Group V. (by £ units). It is impossible to 
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estimate the embarrassment of a. researcher when 
he performes the lengthy and sophisticated 
analysis suggested above, finds significant 
differences, and then interpretes the rela- 
tionship backwards because of a failure to 
identify the adjusted means. 

Test 3: Are there regions where the treatments differ 

in effectiveness (II ^ vs H p )? {This test is 

only employed if F p is statistically signif- 

cant.) This .is the g ener a j i ze d Jobnson-Neyman 
analysis for k group’s, n co variables . 



Calculations: Under the maximum likelihood 

estimate of ~ -1 X J where 1^' is 

the estimate under and X is a specified 
data point. C is an arbitrary set of coefficients 
(C^, 0 ? , . .., C k ) such that C = U" 1 K where 



U = 



X T ^ 1 x' - X t ; 1 X' 



0 

0 

1 



X T ^ 1 X' - X T " 1 x '. . . o 



0 

1 



6 ... X T ] "^ 3 X / - x T^ 1 x 

1 ... 1 1 



K = - £3) - jJ c )> 0], 

Lik'eMee the sum of squares deviation due to H, is 

J 
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= < + £ 
i s i=i 



A fi 

0. X B? and 












k A o . p . 

«: ■-- ? C. X B, where As. is 
a i^j i 1 i 

p p 

distributed *X (k - 1) find As* represents the 
gain in sum of squares deviations due to H.. 

J 

H. vs H can bo tested by 

J R 



3* 



AsyOc - i) 
S^/df 



If P is not significant then the groups 

do not differ significantly at point X« If 

P. is significant then the groups differ sig- 

32S 

nificantly for the data point X, and the 
products X B|, X , X fig should bo 

examined to determine the treatment which will 
maximize performance for X* 

Tho results of the analysis can be used 
effectively to assign individuals, on the basis 
of covariablc scores, to treatments where 
their predicted achievement is the highest. 

In tho aimplist situation when k == n ~ the 
solution reduces to a conic section end can 
be graphed as illustrated in Figure 6, For 
this example the treatment given Group 1 is 



superior for students with 1 ow IQ and initial 
ability vihile the treatment given Group 2 is 
superior for students with high IQ and initial 
ability. Likewise, ns new students enter the 
program they can bo directed to the teaching 
method v.’hich promises the greater success based 
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on their initial ability and IQ* 



FIGURE 6 

EXAMPLE OF A JOHNSON- NJSYMAN SOLUTION 




Use of the Jobnaon-Iic v mn.n Proc c d ure for 
Interaction Ana lyses 

While the Jobnaon-Neyman Procedure might be considered 
an overly complicated substitute for ANCOVA which must be 
resorted to when regression slopes are not homogeneous, it 
con also ho viewed by educators as a powerful tool for 
measuring the relationship between learner characteristics 
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and teaching streegics • There has been a great deal of 
interest in individual differences which affect learning 
performance, but generally the analyses used do not give 
necessary insight into the underlying dynamics. Using a. 
computer to handle the computation, even a 11 non-quanti tati ve 
type” can easily test, describe, and use be Johnson- 
Neyman methodology to classify students by their individual 
needs as well as by the characteristics of teaching methods. 
Hopefully, this kind of classification will solve some of the 
problems which arise because current views of educational 
realities are too simplistic. It may well be that rather 
than a nuisance in ANCOVA, treatmcnt-covariablc interaction 
is the key to understands ng the critical relationship oetue&n 
teaching and learning. 



Summary 



The assumptions underlying eovariable methods were 
analyzed and procedures were suggested for dealing with 
curvilinear! ty, eovariable selection, and nonhonogenei ty of 
regression. The procedure for handling nonhomogeneous group 
regressions was shown to be of value in assigning students to 
various instructional methods on the basis of their Individ- 
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ual character! sties * 



Computer programs which perform the analyses discussed 
in this paper are available upon request from the author. 

An exteiisive bibliography is also available* 
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FOOTNOTES 



An indicated by the title, curvilinear! ty and covariablo 
effectiveness are included as ancillary topics because of 
their considerable impact on the quality of the analysis. 

For the purposes of this paper the term cove.rioto method or 
covariate procedure will refer to a statistical test of 
group differences based on group equations of the form 

* = b 0 + b l X 1 + •*• + W 



Janet D. Elasboff , rr Analysis of Covariance: A 

Delicate Instrument 11 , Amer ican Kducati onal Re search Journal , 

VI (Hay, 1969 ), 383-401, and J ame s"T r . YJ il son amTTkiy 1-. “Cary, 

,f Homogenei ty of Regression--! ts Rationale, Complication end 
Use,” American Educational Research Journal, VI (January, 

1969), W^ToT ~ 

3 

There is considerable controversy surrounding the use 
of covariate methods to analyze quasi-experiments, but this 
docs not change the fact that alternative procedures ore 
even more difficult to apply and have not been proven superior. 
See Donald A, Campbell and Julian Stanley, if Experimental 
and Quasi -experimental Designs for Research on Teaching, 11 
Handbook of Research on Teaching, ed. hv N. L« Gage (Chicago: 
Rand McNally and Company, 1963), pp. l71-?4&* 

^The discussion of these assumptions parallels Elasboff T s 
11 Analysis of Covariance , Ir 
... 

M. Atiqullnh, 11 The Robustness of the Covariance 
Analysis of a One-way Classification, rr Biometriko, IJ{Decomber, 
1964), 365-37?. 

^This is not true if the ,r spread ,r of covariable values 
is small or there are outlying cases* In one situition 
v;here the spread of valuos v;es smell , 50 outliers caused the 
correlations to drop from .7 to .3 where ^0.000 cases were 
used in the analysis. Obviously problems~olf TETs type will 
ruin any covoriablo procedure, 

7 

Independence is guaranteed in a true experiment by the 
random assignment of subjects to groups. 
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^Campbell end St inlay, 11 Experimental and (Jussi- 
Experimental Designs," pp. ?31-234* 

9 

Frederic Lord, "A Paradox in the Interpretation of 
Group Compari sons f " Psychological Bulletin, LXVIII (No* ?) , 
3011-30?. 

^Andrevr Porter , Tho_ Effe ct a o f Using Fall ible Var iables 
in the Analysi s of Covariance OhqvubTx she ph , P . Si s srir ta tTon , 
University of Wi sconsin ) , Ann Arbor. Michigan: University 

Microfilms, 1967, No. 67-12# 

11 

William L, Hays, Stn ti sti es for P sychologists ( Nev: York: 
Holt, Rinehart and Winston? 1965T, pp. 3”39~~?30*. 

12 

Since an infinite number of regression equations can 
be fitted to the data, the exact relationship must arise 
from a careful study of theory rather than statistical 
accidents • 

1 3 

This corresponds to a singular variance -cover! once 
matrix which cannot be inverted to complete the analysis. 

^For example, the new variable could be the sum of the 
norm®! scores for each c ova ri able , 

1 ? 

The derivation here is vnsed on Jersey Neyman’c method 
for testing linear hypotheses as refined by Palmer 0, Johnson. 

See Palmer 0 o Johnson, " Tho Jobnsos-Uoymsn Technique, Its 
Theory and Application," Psvchqmot^ika , XV (December, 19?0), 
349-36?; Palmer- 0. Johnson arid Robert W c H. Jackson, Mo dern 
Stati stlcal Method p ( Chicago : Rand Me Ha 11 y , 19 S9 ) ; PaTmor ~ 

0. Johnson Vrid Cyril Hoyt, " On Determining Three Dimensional 
Regions of Significance, " Jou rna l of E xperiment s! E ducation, 

XV (March, 1967), 203-212; end Palmer oYJrinnaori arid yri 1 
Hoyt, Th o Thoo r y of Idn o a r Hypo the s c a with April c a t i o n a t o 

Educa tional P^ob l oTns Oil nresota: 1‘irii ve r n it? J o f THTmririoTa 

Bureau of Educe t Moira 1 Research, 196?). 

■^Where a variable name is underlined, it will refer to 
a vector or matrix of values. 

1 7 

Percy J). Peckhan, " An Inve^nsigv,tion of the Effects of 
Non-Homogcneous Regression Slopes Upon the F - To 3 t of Analysis 
of Covariance," laboratory of Educational Research Rcro^t, 

No . ] 6 ( Boulder i Colors do: Un i v entity '67 ~oToT y rlTfo? lYntHT 

^The magnitude of the difference is identical for cither 
although the intercepts are generally less intuitively satisfying. 




